Skip to content

Key parquet column options by leaf path so nested columns get compressed - #8674

Open
jayzuccarelli wants to merge 1 commit into
huggingface:mainfrom
jayzuccarelli:fix/issue-8671
Open

jayzuccarelli wants to merge 1 commit into
huggingface:mainfrom
jayzuccarelli:fix/issue-8671

Conversation

@jayzuccarelli

Copy link
Copy Markdown
Contributor

Fixes #8671

Summary

pq.ParquetWriter matches per-column compression / use_dictionary / column_encoding against the leaf column paths (train.list.element.input...), but both ParquetWriter._build_writer and ParquetDatasetWriter._write keyed them by top-level feature name. Nested list/struct columns never matched, so they fell back to pyarrow's defaults: uncompressed and no dictionary encoding.

This moves the options into one get_parquet_column_options(features, schema) helper in arrow_writer.py that expands each column to its leaf paths, and uses it at both sites. Media columns (Image, Audio, ...) are structs too, so their "none"/PLAIN settings only held because unmatched leaves fell back to the defaults. The output for them is unchanged; the settings now actually match those leaves.

Repro from the issue after the fix:

question                                             -> SNAPPY
train.list.element.input.list.element.list.element   -> SNAPPY
train.list.element.output.list.element.list.element  -> SNAPPY

Test

Added test_parquet_writer_compresses_nested_columns in tests/test_arrow_writer.py (flat, nested list/struct, Array2D, Image). It fails on main and passes with the fix. tests/test_arrow_writer.py, tests/io/test_parquet.py, tests/packaged_modules/test_parquet.py and tests/features/test_image.py pass; make quality is clean.

@HuggingFaceDocBuilderDev

Copy link
Copy Markdown

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

push_to_hub() / to_parquet() leave nested list/struct columns uncompressed

2 participants